Repository navigation
refactor(writer): split VortexWriter into SegmentEncoder, SegmentSink, LayoutTree, GlobalDictWriter and ZoneMapWriter (#663) - #683
Merged
Conversation
dfa1
force-pushed
the
refactor-vortex-writer-663
branch
2 times, most recently
from
November 7, 2026 07:03
291e50c to
bdbd549
Compare
dfa1
force-pushed
the
refactor-vortex-writer-663
branch
from
November 7, 2026 07:04
bdbd549 to
aec4df1
Compare
…, LayoutTree, GlobalDictWriter and ZoneMapWriter (#663) VortexWriter was 1772 lines with ~40 fields and no grouping. It keeps the public factories, schema validation, writeChunk, column chunking and repartition, and the close() orchestration (396 lines); five package-private collaborators take the rest: - SegmentEncoder: candidate lists and registries, the edition guard, ColumnEncoding overrides, the dict probe, per-segment encode + stats. Immutable (the caller's encoder list is copied), and the only state the executor tasks touch. - SegmentSink: channel, ordered in-flight window, segment and encoding-id bookkeeping, array/stats FlatBuffers, footer, postscript, trailer. - LayoutTree: the layout tree and the footer's layout_specs it indexes; the only place FbsLayout nodes are built. - GlobalDictWriter: first-batch detection, ingest, budget eviction and demotion (back through a ChunkAppender callback), flush. - ZoneMapWriter: zoned accumulators, vortex.zoned and legacy vortex.stats flush. RowCount.of (writer.encode) replaces VortexWriter.arrayLength, CascadingCompressor's switch and ZoneAccumulator's reflective length. Also dropped redundant state (dictCandidates was dictStates.keySet(); the retained-bytes cache equalled the live sum), one CompletionException unwrap, inline FQNs, stray docs, wrapped ternaries and inline field initializers (every field is assigned in its constructor). No behavior change: 96 files (12 Raincloud Parquet files x 8 option sets incl. legacy edition, cascading 0, forced demotion) are byte-identical to main; writer and integration (vortex-jni) suites pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dfa1
force-pushed
the
refactor-vortex-writer-663
branch
from
November 7, 2026 07:05
aec4df1 to
edde7c8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #663.
VortexWriterwent from 1772 to 396 lines. It keeps the public factories, schema validation,writeChunk, column chunking and repartition, and theclose()orchestration. Five package-private collaborators take the rest:SegmentEncoder: candidate lists and registries, the edition guard,ColumnEncodingoverrides, the dict probe, per-segment encode and stats. It's immutable (the caller's encoder list is copied), and it's the only writer state the executor tasks touch.SegmentSink: the channel, the ordered window of segments being encoded, segment and encoding-id bookkeeping, the array/stats FlatBuffers, footer, postscript and trailer.LayoutTree: the layout tree and the footer'slayout_specslist it indexes into. It's the only placeFbsLayoutnodes are built.GlobalDictWriter: first-batch detection, ingest, budget eviction and demotion (replayed through aChunkAppendercallback), and the flush at close.ZoneMapWriter: the zoned accumulators, and thevortex.zonedand legacyvortex.statsflush.RowCount.of(inwriter.encode) replaces three copies of the row-count logic:VortexWriter.arrayLength, the switch inCascadingCompressorandZoneAccumulator's reflective length.Removed along the way:
dictCandidates(it was alwaysdictStates.keySet()), the retained-bytes cache (always equal to the live sum), a duplicatedCompletionExceptionunwrap, inline fully-qualified names, stray docs, wrapped ternaries and inline field initializers (every field is now assigned in its constructor).Verification
WriteOptionsvariants (default, compact, zone maps off, legacy edition, no editions, forced dict demotion, cascading 0, global dict off). All 96 outputs have the same sha256 as onmain../mvnw verify -pl writer -am: 2576 tests, checkstyle and javadoc clean../mvnw verify -pl integration -am(vortex-jni interop): see the latest comment for the run on the final commit.Found while testing, already present on
mainand not fixed here:WriteOptions.cascading(0)fails on 4 of the 12 files (vortex.masked: no inner encoding for Bool/Struct,Argument is not an array).🤖 Generated with Claude Code